﻿<!DOCTYPE html>
<html>

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Hadoop入门（慕课网：认识Hadoop--基础篇笔记）-不古出品</title>
  <link rel="stylesheet" href="https://stackedit.io/style.css" />
</head>

<body class="stackedit">
  <div class="stackedit__html"><h1><a id="1Hadoop_0"></a>1.初识Hadoop</h1>
<p>大数据时代已经到来，越来越多的行业面临着大量数据需要存储以及分析的挑战。Hadoop，作为一个开源的分布式并行处理平台，以其高扩展、高效率、高可靠等优点，得到越来越广泛的应用。本课旨在培养学员理解Hadoop的架构设计以及掌握Hadoop的运用能力。</p>
<h2><a id="11_2"></a>1.1前言</h2>
<h3><a id="111_3"></a>1.1.1课程名称</h3>
<p>Hadoop大数据平台架构与实践</p>
<h3><a id="112_5"></a>1.1.2主要内容</h3>
<ul>
<li>1.大数据的相关概念</li>
<li>2.Hadoop的架构和运行机制</li>
<li>3.实战：Hadoop的安装和配置</li>
<li>4.实战：Hadoop开发</li>
</ul>
<h3><a id="113_10"></a>1.1.3学习目标</h3>
<ul>
<li>掌握大数据存储与处理技术的原理（理论知识）</li>
<li>掌握Hadoop的使用和开发能力（实践能力）</li>
</ul>
<h3><a id="114__13"></a>1.1.4 课程学习建议</h3>
<ul>
<li>1.结合书本，知识点更加系统全面<br>
对应的书本：hadoop技术详解、hadoop权威指南</li>
<li>2.实践经验很重要，边听课边实践。</li>
</ul>
<h3><a id="115_17"></a>1.1.5课程预备知识：</h3>
<ul>
<li>linux常用命令</li>
<li>java编程基础</li>
</ul>
<h2><a id="12Hadoop_21"></a>1.2Hadoop的前世今生</h2>
<h3><a id="121Hadoop_22"></a>1.2.1Hadoop基本概念</h3>
<p>Hadoop是解决大数据的分布式集成架构。<br>
当数据达到一定规模时，单机的存储和分析就变得非常困难，存储量和效率都无法达到用户的需求。<br>
所以，为了解决大数据的存储和处理，Google提出了三大技术:</p>
<ul>
<li><strong>MapReduce</strong>
<ul>
<li>概念:"Map（映射）“和"Reduce（归约）”，它们的主要思想，都是从函数式编程语言里借来的，还有从矢量编程语言里借来的特性。它极大地方便了编程人员在不会分布式并行编程的情况下，将自己的程序运行在分布式系统上。 当前的软件实现是指定一个Map（映射）函数，用来把一组键值对映射成一组新的键值对，指定并发的Reduce（归约）函数，用来保证所有映射的键值对中的每一个共享相同的键组。</li>
</ul>
</li>
<li><strong>BigTable</strong>
<ul>
<li>是Google设计的分布式数据存储系统，用来处理海量的数据的一种非关系型的数据库</li>
</ul>
</li>
<li><strong>GFS</strong>
<ul>
<li>是一个可扩展的分布式文件系统，用于大型的、分布式的、对大量数据进行访问的应用。它运行于廉价的普通硬件上，并提供容错功能。它可以给大量的用户提供总体性能较高的服务。</li>
</ul>
</li>
</ul>
<p><strong>Hadoop是模仿Google三大技术的开源实现</strong>。相比于Google之前的解决方案，它有如下优势:</p>
<ul>
<li>（1）降低成本，能用PC机就不用大型机和高端存储;</li>
<li>（2）因为用的是PC机，所以经常发生硬件错误，所以通过软件来保证高可靠性；</li>
<li>（3）简化了并行分布式计算。</li>
</ul>
<h3><a id="122Hadoop_38"></a>1.2.2为什么取名Hadoop</h3>
<p>创作者以他儿子一个黄色的玩具小象命名<br>
<img src="https://img-blog.csdnimg.cn/5ba4d074225d46a8938b4a4b11d4da24.png" alt="Hadoop"></p>
<h2><a id="13Hadoop_41"></a>1.3Hadoop的功能与优势</h2>
<h3><a id="131_Hadoop_42"></a>1.3.1 Hadoop是什么</h3>
<p>是一个开源的、分布式存储和分布式计算平台</p>
<h3><a id="132Hadoop_44"></a>1.3.2Hadoop的两个核心组件</h3>
<ul>
<li>1.HDFS，分布式文件系统，存储海量数据。</li>
<li>2.MapReduce，并行处理框架，实现任务分解和调度。</li>
</ul>
<h3><a id="133Hadoopk_47"></a>1.3.3Hadoopk可以做什么</h3>
<p>搭建大型数据仓库，PB级数据的存储、处理、分析、统计等业务。<br>
<img src="https://img-blog.csdnimg.cn/0a73ce30503144adbc55a9f6be6f5b12.png" alt="在这里插入图片描述"></p>
<h3><a id="133Hadoopk_51"></a>1.3.3Hadoopk的优势</h3>
<ul>
<li>1.高扩展，理论上无限扩展</li>
<li>2.低成本</li>
<li>3.成熟的生态圈<br>
<img src="https://img-blog.csdnimg.cn/f8809c3f92714b39973c0b83b7e367d8.png" alt="Hadop生态圈"></li>
</ul>
<h3><a id="134Hadoopk_56"></a>1.3.4Hadoopk的应用</h3>
<h2><a id="14Hadoop_58"></a>1.4Hadoop的生态系统与版本</h2>
<p><img src="https://img-blog.csdnimg.cn/40415058443b42958fd20dcfdb3b662f.png" alt="应用在这里插入图片描述"></p>
<h3><a id="141hadoop_60"></a>1.4.1hadoop生态系统</h3>
<ul>
<li>1.hdfs分布式存储系统</li>
<li>2.mapreduce 大数据编程模型</li>
<li>3.相关开源工具：
<ul>
<li>（1）hive：将sql语句转化为hadoop任务，降低使用hadoop的门槛</li>
<li>（2）HBASE：存储结构化数据的分布式数据库，放弃事务特性，追求更高的扩展，它提供数据的随机读写和实时访问，实现对表数据的读写功能</li>
</ul>
<pre><code>注：和传统的关系型数据库的区别是放弃事务特性，追求更高的扩展、和HDFS的区别就是habse提供数据的随机读写和实时访问，实现对表数据的读写功能
</code></pre>
<ul>
<li>（3）zookeeper:监控Hadoop集群里的每个节点的状态，管理整个集群的配置，维护数据节点之间的一致性</li>
</ul>
</li>
</ul>
<h3><a id="142hadoop_70"></a>1.4.2hadoop版本</h3>
<ul>
<li>1.x：稳定</li>
<li>2.x：不稳定</li>
</ul>
<h1><a id="2Hadoop_73"></a>2.Hadoop的安装</h1>
<ul>
<li>Step1:准备liunx环境；</li>
<li>Step1:安装JDK；</li>
<li>Step1:配置hadoop；</li>
</ul>
<h2><a id="21liunx_77"></a>2.1准备liunx环境</h2>
<p>两种方式：</p>
<ul>
<li>1.本机先安装虚拟机，再安装linux</li>
<li>2.租用云主机，申请公网IP</li>
</ul>
<h2><a id="22JDK_81"></a>2.2安装JDK</h2>
<p>Linux 下下载安装JDK （centos）</p>
<pre><code class="prism language-javascript">
<span class="token comment">//1、下载  ： </span>
yum install java<span class="token operator">-</span><span class="token number">1.7</span><span class="token number">.0</span><span class="token operator">-</span>openjdk
<span class="token comment">//查看可升级下载的软件包</span>
yum search java<span class="token operator">|</span>grep jdk

<span class="token comment">//2、配置环境变量： </span>
vim <span class="token operator">/</span>etc<span class="token operator">/</span>profile
<span class="token keyword">export</span> <span class="token constant">JAVA_HOME</span><span class="token operator">=</span><span class="token operator">/</span>usr<span class="token operator">/</span>lib<span class="token operator">/</span>jvm<span class="token operator">/</span>java<span class="token operator">-</span><span class="token number">7</span><span class="token operator">-</span>openjdk<span class="token operator">-</span>amd64
<span class="token keyword">export</span> <span class="token constant">JRE_HOME</span><span class="token operator">=</span>$<span class="token constant">JAVA_HOME</span><span class="token operator">/</span>jre
<span class="token keyword">export</span> <span class="token constant">CLASSPATH</span><span class="token operator">=</span><span class="token punctuation">.</span><span class="token operator">:</span>$<span class="token constant">JAVA_HOME</span><span class="token operator">/</span>lib<span class="token operator">:</span>$<span class="token constant">JRE_HOME</span><span class="token operator">/</span>lib<span class="token operator">:</span>$<span class="token constant">CLASSPATH</span>
<span class="token keyword">export</span> <span class="token constant">PATH</span><span class="token operator">=</span>$<span class="token constant">JAVA_HOME</span><span class="token operator">/</span>bin<span class="token operator">:</span>$<span class="token constant">JRE_HOME</span><span class="token operator">/</span>bin<span class="token operator">:</span>$<span class="token constant">PATH</span>  

<span class="token comment">//3、让配置生效</span>
source <span class="token operator">/</span>etc<span class="token operator">/</span>profile

<span class="token comment">//4、测试 </span>
java <span class="token operator">-</span>version

</code></pre>
<pre><code>注解：
1.Linux下用冒号 : 分割路径
2.$PATH / $JAVA_HOME / $JRE_HOME 是用来引用原来环境变量的值，在设置环境变量时不能把原来的值给覆盖掉
3.$CLASSPATH 中 当前目录 “.”不能丢
4.export 是把这4个变量导出为全局变量
</code></pre>
<h2><a id="23Hadoop_111"></a>2.3配置Hadoop</h2>
<h3><a id="231_112"></a>2.3.1安装总结</h3>
<ul>
<li>1.下载hdoop按照包并进行解压</li>
<li>2.配置hdoop-env.sh文件，目的是配置jdk，并在profile配置haddoop的安装位置</li>
<li>3.配置core-site.xml文件：hdoop的核心文件，里面有关于hdoop的节点端口与主机端口</li>
<li>4.配置hdfs-site.xml文件：hdoop的文件存储的基本信息与目录</li>
<li>5.配置mapred-site.xml文件：hadoop的计算节点的端口号</li>
<li>6.启动hadoop：start-all.sh</li>
<li>7.查看端口：jps,可以看到五大守护进程说明正确</li>
<li>8.停止hdoop：stop-all.sh</li>
</ul>
<h3><a id="232_121"></a>2.3.2安装细节</h3>
<ul>
<li>1、下载Hadoop安装包 ，两种方式
<ul>
<li>方式一：官网下载好后，通过Xftp上传到Linux上，下载地址http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz</li>
<li>方式一：通过 wget命令直接下载到Linux服务器上<br>
$ wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz</li>
</ul>
</li>
<li>2、解压到指定目录下；
<ul>
<li>移动文件：<br>
$ mv 文件 /opt</li>
<li>解压：<br>
$ tar -zxvf hadoop-1.2.1.tar.gz</li>
</ul>
</li>
<li>3、配置hadoop-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml四个文件</li>
</ul>
<pre><code class="prism language-javascript"><span class="token comment">//1、hadoop-env.sh</span>
配置java 环境变量的地址
<span class="token keyword">export</span> <span class="token constant">JAVA_HOME</span><span class="token operator">=</span><span class="token operator">/</span>usr<span class="token operator">/</span>lib<span class="token operator">/</span>jvm<span class="token operator">/</span>java<span class="token operator">-</span><span class="token number">7</span><span class="token operator">-</span>openjdk<span class="token operator">-</span>amd64

<span class="token comment">//2、 core-site.xml</span>
<span class="token operator">&lt;</span>configuration<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>name<span class="token operator">&gt;</span>hadoop<span class="token punctuation">.</span>tmp<span class="token punctuation">.</span>dir<span class="token operator">&lt;</span><span class="token operator">/</span>name<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>value<span class="token operator">&gt;</span><span class="token operator">/</span>hadoop<span class="token operator">&lt;</span><span class="token operator">/</span>value<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>configuration<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>name<span class="token operator">&gt;</span>dfs<span class="token punctuation">.</span>name<span class="token punctuation">.</span>dir<span class="token operator">&lt;</span><span class="token operator">/</span>name<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>value<span class="token operator">&gt;</span><span class="token operator">/</span>hadoop<span class="token operator">/</span>name<span class="token operator">&lt;</span><span class="token operator">/</span>value<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>name<span class="token operator">&gt;</span>fs<span class="token punctuation">.</span>default<span class="token punctuation">.</span>name<span class="token operator">&lt;</span><span class="token operator">/</span>name<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>value<span class="token operator">&gt;</span>hdfs<span class="token operator">:</span><span class="token operator">/</span><span class="token operator">/</span>imooc<span class="token operator">:</span><span class="token number">9000</span><span class="token operator">&lt;</span><span class="token operator">/</span>value<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>property<span class="token operator">&gt;</span>

<span class="token comment">//3、hdfs-site.xml配置</span>
<span class="token operator">&lt;</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>name<span class="token operator">&gt;</span>dfs<span class="token punctuation">.</span>data<span class="token punctuation">.</span>dir<span class="token operator">&lt;</span><span class="token operator">/</span>name<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>value<span class="token operator">&gt;</span><span class="token operator">/</span>hadoop<span class="token operator">/</span>data<span class="token operator">&lt;</span><span class="token operator">/</span>value<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>property<span class="token operator">&gt;</span>

<span class="token comment">//4、mapred-site.xml配置</span>
<span class="token operator">&lt;</span>property<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>name<span class="token operator">&gt;</span>mapred<span class="token punctuation">.</span>job<span class="token punctuation">.</span>tracker<span class="token operator">&lt;</span><span class="token operator">/</span>name<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span>value<span class="token operator">&gt;</span>imooc<span class="token operator">:</span><span class="token number">9001</span><span class="token operator">&lt;</span><span class="token operator">/</span>value<span class="token operator">&gt;</span>
<span class="token operator">&lt;</span><span class="token operator">/</span>property<span class="token operator">&gt;</span>

</code></pre>
<ul>
<li>4、编辑/etc/profile文件，配置hadoop相关的环境变量</li>
</ul>
<pre><code class="prism language-javascript">vim <span class="token operator">/</span>etc<span class="token operator">/</span>profile
添加hadoop的<span class="token constant">HADOOP_PATH</span>
<span class="token keyword">export</span> <span class="token constant">HADOOP_PATH</span><span class="token operator">=</span><span class="token string">"安装目录"</span>
<span class="token constant">PATH</span>里面添加$<span class="token constant">HADOOP_HOME</span><span class="token operator">/</span>bin<span class="token operator">:</span>$<span class="token constant">PATH</span>
<span class="token comment">//让配置生效</span>
source <span class="token operator">/</span>etc<span class="token operator">/</span>profile
<span class="token comment">//测试是否安装成功</span>
hadoop
</code></pre>
<ul>
<li>5、第一次使用hadoop先进行格式化</li>
</ul>
<pre><code class="prism language-javascript"><span class="token comment">//对namenode 进行格式化</span>
$ hadoop namenode <span class="token operator">-</span>format；
</code></pre>
<ul>
<li>6、启动hadoop<br>
start-all.sh</li>
<li>7、检查进程：jps<img src="https://img-blog.csdnimg.cn/37e4ee4e1126403d8d3f8647339b6e8b.png" alt="jps"></li>
</ul>
<pre><code class="prism language-javascript"><span class="token comment">// 查看hadoop文件系统下下有哪些文件</span>
hadoop fs <span class="token operator">-</span>ls
</code></pre>
<h1><a id="3HadoopHDFS_189"></a>3.Hadoop的核心-HDFS简介</h1>
<h2><a id="31HDFS_190"></a>3.1HDFS基本概念</h2>
<ul>
<li>HDFS设计架构：
<ul>
<li>块:文件以“块”进行存储，HDFS默认块大小为64M</li>
<li>namenode：是管理节点，存放文件元数据，包括文件与数据块的映射表、数据块与数据节点的映射表</li>
<li>datenode：工作节点，真正存储数据块<br>
最终体系结构：由block、namenode、datanode、secondarynamenode、client组成</li>
</ul>
</li>
</ul>
<p>HDFS的文件被分成块进行存储，HDFS块默认大小是64MB，块是整个文件存储处理的逻辑单元。</p>
<p>HDFS最终体系结构：由block、namenode、datanode、secondarynamenode、client组成。<br>
<img src="https://img-blog.csdnimg.cn/1d241bc9d56f4c51be4bb7ff09639932.png" alt="HDFS架构图"></p>
<h2><a id="32HDFS_201"></a>3.2HDFS数据管理与容错</h2>
<p>为保证硬件上的容错，数据块有多份冗余。</p>
<ul>
<li>数据块副本：每个数据块3个副本，分布在2机架3节点上（容错性）</li>
<li>心跳检测：DataNode定期向NameNode发心跳消息。</li>
<li>二级NameNode：NameNdoe定期同步元数据映像文件到二级NameNode（secondryNameNode）,一旦故障，备胎转正。</li>
</ul>
<h2><a id="32HDFS_207"></a>3.2HDFS读写文件的流程</h2>
<ul>
<li>HDFS读取文件的流程：
<ul>
<li>（1）客户端向namenode发起独立请求，把文件名，路径告诉namenode；</li>
<li>（2）namenode查询元数据，并把数据库返回客户端；</li>
<li>（3）此时客户端就明白文件包含哪些块，这些块在哪些datanode中可以找到；<br>
<img src="https://img-blog.csdnimg.cn/031a2282b535412db6c51e129f835c9f.png" alt="HDFS读取文件的流程"></li>
</ul>
</li>
<li>HDFS写文件流程：
<ul>
<li>（1）客户端把文件拆分成固定大小64M的块，并通知namenode；</li>
<li>（2）namenode找到可用的datanode返回给客户端；</li>
<li>（3）客户端根据返回的datanode，对块进行写入；</li>
<li>（4）通过流水线管道流水线复制；</li>
<li>（5）更新元数据。告诉namenode已经完成了创建心的数据块，保证了namenode中的元数据都是最新的状态。<br>
<img src="https://img-blog.csdnimg.cn/c70e33267a2f4bcab4d332dc95569bd1.png" alt="HDFS写文件流程"></li>
</ul>
</li>
</ul>
<h2><a id="32HDFS_220"></a>3.2HDFS的特点</h2>
<ul>
<li>HDFS特点：
<ul>
<li>1、数据冗余，硬件容错（一式三份来保证）。</li>
<li>2、流式数据访问：写一次，读多次，一旦写入无法修改，只能通过写入到新的块删除旧文件。</li>
<li>3、存储大文件（特适合，因为小文件多，势必加重NameNode的负担）。</li>
</ul>
</li>
<li>HDFS适用性及局限性：
<ul>
<li>1、适合数据批量读写，吞吐量高；</li>
<li>2、不适合交互式应用、低延迟很难满足；</li>
<li>3、适合一次写入多次读取、顺序读写；</li>
<li>4、不支持多用户并发写相同文件。</li>
</ul>
</li>
<li>HDFS优缺点：
<ul>
<li>优点：存储块大，吞吐量高，为存储大文件设计；</li>
<li>缺点：延迟高，不适合交互式访问，不支持多用户同时操作一个块。</li>
</ul>
</li>
</ul>
<h2><a id="33HDFS_233"></a>3.3HDFS的使用</h2>
<ul>
<li>提供了 shell 接口，可以进行命令行操作</li>
</ul>
<pre><code class="prism language-javascript">hadoop namenode <span class="token operator">-</span>format	#格式化namenode
hadoop fs <span class="token operator">-</span>ls <span class="token operator">/</span> #打印 <span class="token operator">/</span> 目录文件列表
hadoop fs <span class="token operator">-</span>mkdir input #创建目录 input
hadoop fs <span class="token operator">-</span>put hadoop<span class="token operator">-</span>env<span class="token punctuation">.</span>sh input<span class="token operator">/</span> #上传文件 hadoop<span class="token operator">-</span>env<span class="token punctuation">.</span>sh 到 input 目录下
hadoop fs <span class="token operator">-</span><span class="token keyword">get</span> input<span class="token operator">/</span>abc<span class="token punctuation">.</span>sh hadoop<span class="token operator">-</span>envcomp<span class="token punctuation">.</span>sh #从 input 目录中下载文件
hadoop fs <span class="token operator">-</span>cat input<span class="token operator">/</span>hadoop<span class="token operator">-</span>env<span class="token punctuation">.</span>sh #查看文件 input<span class="token operator">/</span>hadoop<span class="token operator">-</span>env<span class="token punctuation">.</span>sh 
hadoop dfsadmin <span class="token operator">-</span>report #dfs报告
</code></pre>
<h1><a id="4HadoopMapReduce_244"></a>4.Hadoop的核心-MapReduce原理与实现</h1>
<h2><a id="41MapReduce_245"></a>4.1MapReduce的原理</h2>
<ul>
<li>MapReduce原理：分而治之，一个大任务分成多个子任务（map），并行执行之后，合并结果（reduce）。</li>
</ul>
<pre><code>eg：做统计的时候，把统计的文件拆分，然后分别统计每一个数据出现的次数，然后合并拆分项，就可以统计每一个数据出现的总次数。
</code></pre>
<ul>
<li>
<p>MapReduce处理数据过程主要分成2个阶段：Map阶段和Reduce阶段。首先执行Map阶段，再执行Reduce阶段。Map和Reduce的处理逻辑由用户自定义实现，但要符合MapReduce框架的约定。</p>
</li>
<li>
<p>正式执行Map前，需要将输入数据进行”分片”。所谓分片，就是将输入数据切分为大小相等的数据块，每一块作为单个Map Worker的输入被处理，以便于多个Map Worker同时工作。分片完毕后，多个Map Worker就可以同时工作了。每个Map Worker在读入各自的数据后，进行计算处理，最终输出给Reduce。Map Worker在输出数据时，需要为每一条输出数据指定一个Key。这个Key值决定了这条数据将会被发送给哪一个Reduce Worker。Key值和Reduce Worker是多对一的关系，具有相同Key的数据会被发送给同一个Reduce Worker，单个Reduce Worker有可能会接收到多个Key值的数据。</p>
</li>
<li>
<p>在进入Reduce阶段之前，MapReduce框架会对数据按照Key值排序，使得具有相同Key的数据彼此相邻。如果用户指定了”合并操作”(Combiner)，框架会调用Combiner，将具有相同Key的数据进行聚合。Combiner的逻辑可以由用户自定义实现。这部分的处理通常也叫做”洗牌”(Shuffle)。<br>
接下来进入Reduce阶段。相同的Key的数据会到达同一个Reduce Worker。同一个Reduce Worker会接收来自多个Map Worker的数据。每个Reduce Worker会对Key相同的多个数据进行Reduce操作。最后，一个Key的多条数据经过Reduce的作用后，将变成了一个值。<br>
<img src="https://img-blog.csdnimg.cn/c068d67c33f248749625a5096ef4b639.png" alt="MapReduce原理"></p>
</li>
</ul>
<h2><a id="42MapReduce_256"></a>4.2MapReduce的运行流程</h2>
<p><img src="https://img-blog.csdnimg.cn/eb9e258d8b864f738755c4e08b1b028e.png" alt="流程"></p>
<h3><a id="421__259"></a>4.2.1. 原理</h3>
<ul>
<li>分而治之 的思想，一个大任务分成多个小任务（map）,并行执行后，合并结果（reduce）.</li>
</ul>
<h3><a id="422__262"></a>4.2.2 运行流程</h3>
<p><img src="https://img-blog.csdnimg.cn/f43931a951734a40849f3b8960d7ffad.png" alt="执行过程"></p>
<ul>
<li>基本概念：
<ul>
<li>Job &amp; Task：<br>
一个 Job（任务、作业） 被切分为多个 Task，Task 又分为 MapTask 和 ReduceTask</li>
<li>JobTracker<br>
作业调度<br>
分配任务、监控任务<br>
监控 TaskTracker 的状态</li>
<li>TaskTracker<br>
执行任务<br>
向 JobTracker 汇报任务状态</li>
</ul>
</li>
</ul>
<h3><a id="423__275"></a>4.2.3 容错机制</h3>
<ul>
<li>重复执行：<br>
默认重复执行 4 次，若还是失败，则放弃执行</li>
<li>推测执行：<br>
可以保证任务不会因为某1-2个机器错误或故障而导致整体效率下降</li>
</ul>
<h1><a id="5Hadoop_280"></a>5.开发Hadoop应用程序</h1>
<h2><a id="51WordCount_281"></a>5.1WordCount单词计数（上）</h2>
<ul>
<li>计算文件中出现每个单词的频数，输入结果按照字母顺序进行排序。利用mapReduce的思想，来计算输入的单词的个数。</li>
<li>Map过程：分而治之</li>
<li>Reduce过程：合并<br>
<img src="https://img-blog.csdnimg.cn/07857956f098430ab9cf7a4a66a4fd0b.png" alt="Map过程"><img src="https://img-blog.csdnimg.cn/8f906777560b4a5abb56ccdc82c653c4.png" alt="Reduce过程"></li>
</ul>
<h2><a id="52WordCount_286"></a>5.2WordCount单词计数（中）</h2>
<ul>
<li>分析源代码</li>
<li>代码其实在hadoop的安装目录下有example，一般的目录是/hadoop/src/examples/org/apache/hadoop/examples/WordCount.jar</li>
<li>源代码  http://hadoop.apache.org/docs/r1.0.4/cn/mapred_tutorial.html</li>
</ul>
<h2><a id="53WordCount_290"></a>5.3WordCount单词计数（下）</h2>
<h3><a id="531_291"></a>5.3.1整体流程</h3>
<ul>
<li>1.编写WordCount.java,包含Mapper类和Reducec类；</li>
<li>2.编译WordCount.java,java -classpath ；</li>
<li>3.打包 jar -cvf WordCount.jar classes/*；</li>
<li>4.作业提交 hadoop jar WordCount.jar WordCount input output；</li>
<li>提交到hadoop中运行，指定输入文件 ，指定输出文件。</li>
</ul>
<h3><a id="532_297"></a>5.3.2具体流程</h3>
<pre><code class="prism language-javascript">
<span class="token comment">//（1）启动hadoop</span>
 start <span class="token operator">-</span>all<span class="token punctuation">.</span>sh
 
<span class="token comment">//（2）rz把某某.java类放到根目录下/opt/根目录下边或者根目录下任意文件</span>
cd <span class="token operator">/</span>mkdir project_hadoop<span class="token operator">/</span>
rz 类路径添加

<span class="token comment">//（3）cd 新创建的文件,创建file1和file2</span>
mkdir input
vi file1
vi file2
ls project_hadoop

<span class="token comment">//（4）创建文件</span>
 hadoop fs <span class="token operator">-</span>mkdir input_wordcount

<span class="token comment">//（5）创建目录：</span>
hadoop fs <span class="token operator">-</span>mkdir input
<span class="token comment">//查看文件：</span>
 hadoop fs <span class="token operator">-</span>ls     
 hadoop fs <span class="token operator">-</span>ls input_wordcount 
 
<span class="token comment">//（6）把file1和file2文件放到input_wordcount</span>
提交输入文件给hadoop
hadoop fs <span class="token operator">-</span>put 文件路径 提交后的路径
<span class="token comment">//例：</span>
hadoop fs <span class="token operator">-</span>put input<span class="token operator">/</span>input_wordcount<span class="token operator">/</span>

<span class="token comment">//（7）查看文件</span>
hadoop fs <span class="token operator">-</span>ls input_wordcount
fs <span class="token operator">-</span>cat input_wordcount<span class="token operator">/</span>file1
fs <span class="token operator">-</span>cat input_wordcount<span class="token operator">/</span>file2

<span class="token comment">//（8）编译java文件</span>
javac <span class="token operator">-</span>classpath <span class="token operator">/</span>opt<span class="token operator">/</span>hadoop<span class="token operator">-</span><span class="token number">1.2</span><span class="token number">.1</span><span class="token operator">/</span>hadoop<span class="token operator">-</span>core<span class="token operator">-</span><span class="token number">1.2</span><span class="token number">.1</span><span class="token punctuation">.</span>jar<span class="token operator">:</span><span class="token operator">/</span>opt<span class="token operator">/</span>hadoop<span class="token operator">-</span><span class="token number">1.2</span><span class="token number">.1</span><span class="token operator">/</span>lib<span class="token operator">/</span>commons<span class="token operator">-</span>cli<span class="token operator">-</span><span class="token number">1.2</span><span class="token punctuation">.</span>jar <span class="token operator">-</span>d 编译后地址 编译文件

<span class="token comment">//（9）打包指令</span>
jar <span class="token operator">-</span>cvf 打包后文件名<span class="token punctuation">.</span>jar 某某<span class="token punctuation">.</span>class
<span class="token class-name">jar</span> <span class="token operator">-</span>cvf wordcount<span class="token punctuation">.</span>jar <span class="token operator">*</span><span class="token punctuation">.</span>class

<span class="token comment">//（10）提交jar给hadoop执行</span>
hadoop jar jar包路径 <span class="token function">执行的主函数名</span><span class="token punctuation">(</span>主类名，main方法所在类名<span class="token punctuation">)</span> 输入目录名 输出目录名
<span class="token comment">//例：</span>
hadoop jar project_hadoop<span class="token operator">/</span>wordcount<span class="token punctuation">.</span>jar WordCount input_wordcount output_wordcount

<span class="token comment">//（11）查看通过算法计算出单词个数的结果</span>
fs <span class="token operator">-</span>cat output_wordcount<span class="token operator">/</span>part<span class="token operator">-</span>r<span class="token operator">-</span><span class="token number">00000</span>

</code></pre>
<h2><a id="54MapReduce_349"></a>5.4利用MapReduce进行排序（上）</h2>
<ul>
<li>先分片，再排序<br>
<img src="https://img-blog.csdnimg.cn/6611b3276369451f93d2767a6f6cf3e4.png" alt="分片排序"></li>
</ul>
<h2><a id="55MapReduce_352"></a>5.5利用MapReduce进行排序（下）</h2>
<ul>
<li>分析代码<br>
<img src="https://img-blog.csdnimg.cn/5a2d8e3e5bd846b4a81646f8f489cfa4.png" alt="在这里插入图片描述"><br>
<strong>注：配合慕课网视频使用更佳</strong><br>
链接: <a href="https://www.imooc.com/learn/391">点击跳转</a>.</li>
</ul>
</div>
</body>

</html>
